TRAVELING WAVE SOLUTIONS FOR A PREDATOR-PREY SYSTEM 
WITH SIGMOIDAL RESPONSE FUNCTION 



XIAOBIAO LIN, PEIXUAN WENG, AND CHUFEN WU 

Abstract. We study the existence of traveling wave solutions for a diffusive predator- 
prey system. The system considered in this paper is governed by a Sigmoidal response 

O 

^SJ ' function which is more general than those studied previously. Our method is an im- 

provement to the original method introduced in the work of Dunbar [2l[3]. A bounded 
Wazewski set is used in this work while unbounded Wazewski sets were used in [2] [3]. 



00 



The existence of traveling wave solutions connecting two equilibria is established by us- 
ing the original Wazewski's theorem which is much simpler than the extended version in 
Dunbar's work. 



1.. Introduction 



Predator-prey models are important tools that help us to understand the bio and ecosys- 

> 

' tems surrounding us [9]. An important aspect of the model is how the predator interacts 

. with the prey [SJ [H], which can be described by a functional response that specifies the 

If^ ' rate of feeding of predator upon prey as a function of the prey density. 

o ' 

The existence of traveling wave solutions is an interesting feature in population dynamics 
and plays an important role in understanding the long time asymptotic properties of such 
^ ' systems. In his pioneering works [2l[3l[3], Dunbar obtained the existence of several kinds 

. of traveling wave solutions for diffusive predator-prey systems with type I and type II 

functional responses. He considered the existence of small amplitude periodic traveling 
waves, and "heteroclinic traveling waves" that correspond to heteroclinic orbits connecting 
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equilibrium-to-equilibrium or equilibrium-to-periodic orbits. The methods used by Dunbar 
include the invariant manifold theory, the shooting method, Hopf bifurcation analysis, and 
LaSalle's invariance principle. Huang et al. [llj extended the work in [3j to by using 
Dunbar's method in [4J. An interesting question is whether those results can be extended 
to a system with type III functional response. Recently, Li et al. [13] proved the existence 
of traveling waves in a diffusive predator-prey system with a simplified type III functional 
response ip{u) = i^^^i by employing the method similar to that used in [21 [3]. 

The shooting method used by Dunbar is based on a variant of Wazewski's theorem 
[21 [2111]. Similar to the original Wazewski set, Dunbar's Wazewski set W has the property 
that there is an orbit starting from the unstable manifold of an equilibrium and staying 
in W in the future. However, the Wazewski set W constructed in [21[3ll3] was unbounded. 
In order to ensure the boundedness of the orbit, several additional Lemmas (see Lemma 
7 in [2], Lemma 10 in [3j, Lemma 15 in [3]) were proved to rule out the possibility that 
the constructed orbit may escape to infinity. Although the topological idea of shooting 
method in [21l3l[l] is clear and elegant, the use of unbounded sets W in M'^ or makes the 
argument long and hard to read. Therefore, it is highly desirable to construct a simple, 
bounded Wazewski set W and use the original Wazewski's theorem to simplify the proof 
of the existence of heteroclinic traveling waves related to various predator-prey systems. 

Another improvement is that in the proof of the existence of type I traveling waves, we 
are able to treat the two cases c > c* and c = c* together, while in Dunbar's original work, 
c = was treated by taking a sequence c„ > c,,,, c„ — ?■ c* and looking for convergence 
of the sequence of traveling wave solutions. Finally, let 7 = ^, which is the quotient of 
the predator natural death rate verses the prey growth rate. We have obtained an exact 
threshold value 7*. If 7 > 7*, the traveling waves are oscillatory. It means if 7 is big, 
then the reason the limit of the wave exists is due to the balance of large death rate of 
predator compensated by the large growth rate of prey so the population can approach 
the interior equilibrium. This causes the greater rate of exchange of bio-mass between the 
predator and prey, therefore causes the oscillation. While if < 7 < 7*, the traveling 
waves are non-oscillatory. It means the predators possess enough food to consume or have 
low death rate and then leads to the population density monotonically increasing. In 
Dunbar's article, only a lower estimate for the threshold is given. 
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Consider the following diffusive predator-prey system 



dNi 




N2 



(1.1) 



dt 



d^N2 



ai + 5iiVi + iVf 



dN2 



D2 



dt 



where A'^i and are the population densities of the prey and predator respectively. The 
functional response of the system is more general and biologically useful than those in 
previous works. To the best of our knowledge, no rigorous work has been done on the 
existence of traveling wave solutions for system (jl.ip . 

The shooting technique has been an important method in proving the existence of 
traveling waves solutions, e.g., see [2l[3l[6l[7l[TTl[13] and the references therein. The method 
used in this paper is motivated by the techniques used there. However, as mentioned above, 
we shall construct a bounded set W to replace the unbounded Wazewski set introduced 
in [21 [31 [TTl 113] . Assume that our system has 3 equilibria Ei and E* . Instead of 
the extended Wazewski's theorem used in O [3], we shall use the original Wazewski's 
topological principle as in [8] to prove the existence of an orbit connecting Ei and E* 
in W. We first exam the fiow on the surfaces of W to determine its egress sets. Then 
we construct a curve C on the two dimensional unstable manifold with two endpoints in 
two disjoint egress sets. We show that there is a point on C n W such that the orbit 
started from this point will remain in W by using Theorem 2.1 in Hartman [8, pp. 279]. A 
Liapunov function is constructed to prove that any orbit that remains in W shall approach 
an invariant set in W while the maximal invariant set in W is E* . Thus the solution that 
stays in W will approach the positive equilibrium E* . In other words, it is a heteroclinic 
orbit connecting the unstable equilibrium Ei to E* . We also show the existence of traveling 
wave solution connecting E to E* . Our method is more straightforward than the shooting 
technique used in [21 O [TTl E] • 

In this paper we assume Di = 0. It is the limiting case where the prey species diffuses 
much slower than the predator species, e.g., a plant species being consumed by a rela- 
tively mobile herbivore. The case Di is nonzero and small can be handle by the singular 
perturbation method and will appear in a separate paper (Wu et al. in preparation). 
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To write system (11. Ih in a non-dimensional form, we rescale the variables 
A-^ B-^ U-^ W-^ t'-rt x'- f^x 
By dropping the primes on t, x for notational convenience, (jl.ip becomes 

Uil-U) 



dt ' ' A + BU + m' 

dW d'^W W ( 
+ 



dt 9x2 r yA^BU^V^ 

where r,e,a > 0,^ > 0,5 > 0. Let /3 = f . We require that /3>l + yl + 5>l, 
which ensures (|1.2|) has a positive equilibrium corresponding to the coexistence of the two 
species. It is easy to see that system (jl.2p has three spatially constant equilibria given by 



E(0,0), -El (1,0) and E*(u*,nr), 
where u = — ^ , w = 



2(/3 - 1) 

By using the techniques described above, we will establish the existence of traveling wave 
solutions of system (11.20 connecting the equilibria Ei and E* (type I wave), which is 
called the "waves of invasion" (see Chow and Tam [1], Shigesada and Kawasaki|16j). 
This is of ecological interest since it corresponds to a situation where an environment is 
initially inhabited only by the prey species at its carrying capacity, the small invasion of 
the predators drives the system to a new stable state of co-existence of both species. We 
also establish the existence of traveling wave solutions of system (jl.2p connecting E and 
E* (type II wave). This describes simultaneous spread of predator and prey into a new 
environment. Moreover, a threshold property is given for the two types of traveling waves 
to be oscillatory or non-oscillatory. See Theorem 12.11 for details. 

The organization of the paper is as follows. In Section 2, we state our main results on 
the existence of traveling wave solutions. Section 3 and Section 4 are devoted to proving 
the existence or non-existence for type I and type II waves, respectively. The construction 
of the Wazewski set W will also be given in Sections 3 and 4. In section 5, we discuss the 
threshold property for the oscillation of the traveling waves in terms of 7, the quotient of 
the predator natural death rate verses the prey growth rate. 



traveling wave solutions for a predator-prey system 5 
2.. Main results 

A traveling wave solution of (jl.2p is a solution of the special form U{t, x) = u{x + ct) = 
u{s) and W{t,x) = w{x + ct) = w{s), where s = x + ct and c > is the wave speed. 
Substituting this solution into (|1.2p . we have the wave system 



cu'{s) = u{l — u) 



, , A + Bu + v?' 

(2-1) . 2 

au 



cw'(s) = w"(s) + — ( - 
r \. 



A + Bu + u"^ 

Note that (j2.ip also has three equilibria E, Ei and E* , where E corresponds to the absence 

of both species, Ei corresponds to the prey at the environment carrying capacity in the 

absence of the predator, and E* corresponds to the co-existence of both species. Rewrite 

system (j2.ip as an equivalent system in 

, If uw 
u [s) = -u I 1 — u 



c V A + Bu + u'^ 

(2.2) = z, 



z'{s) = CZ + JW ( 1 



A + Bu + u^J ' 

where /3 = ^, 7 = ^. Then E, Ei and E* correspond to the critical points (0, 0, 0), (1, 0, 0) 
and (ii*,t(;*,0) of ()2.2p respectively. In what follows, we shall still use E,Ei and E* to 
denote (0,0,0), (1,0,0) and {u*,w*,0). The Jacobian matrix of (12. 2p takes the form 

/Vl '^i/'l (2A+Bu)uw u'^ rA 



(2.3) 



1 

. l3'y{2A+Bu)uw fiu^ \ , 

\ {A+Bu+u^y^ ' I ^ A+Bu+u-^ I ^/ 



We call the nonnegative solutions of (12. 2p satisfying 

(2.4) lim (u(s),u;(s)) = (1,0), lim {u{s),w{s)) = (u* ,w*) 

s— 00 s— >+oo 

type I waves, and the nonnegative solutions of (j2.2p satisfying 

(2.5) lim (u(s),u;(s)) = (0,0), lim {u{s),w{s)) = {u* ,w*) 

s— >— 00 s— >+oo 

type II waves. 

Consider the prey isocline of ()2.ip 



, , , (1 - u)(A + Bu + u'^) 
w = h{u) := u e (0, +00). 
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It is easy to check that h'{u) = where 

k{u) := -2u^ + (1 - B)u'^ -A, u G (0, +oo). 

If B > 1, then h{u) is monotone decreasing. Next, we consider the other case < B < 1. 
Obviously, there is a negative real root to k{u) = 0. Let P := (1 — B)'^, Q := —18A, R := 
3^(1 - B), then A := _ 4pji = 12A[27A - (1 - Bf]. (i) If 27A > (1 - B)^, there is a 
complex conjugate pair of roots with positive real part to k{u) = 0. Hence, k{u) < for all 
u G (0, +oo). (n) If 27 A = (1 — B)^, there is a double positive root u = to k{u) = 0. 
It follows that is a local maximum point of k{u) = and thus k{u) < for all 
u G (0, +oo). (i) and (ii) imply that h{u) is monotone decreasing, (iii) If 27A < (1 — B)^, 
there is three distinct real roots to k{u) = 0. Note that k'{u) = 2u{l — B 3n), then 
k{u) = has two positive roots ai,a2 satisfying < ai < < 02 < 1 (see Fig. 1). We 
now state the following result. 




Figure 1. The graph of 
k{u) for < 5 < 1 and 
27^1 < (1 - B)3 




Uo «! Q!2 Ui 1 

Figure 2. The prey iso- 
cline w = h{u) for < 
B < 1 and 27 A < (1 - 
Bf 



Lemma 2.1. If < B < 1 and 27 A < (1 — B)^ , then k{u) = has two positive roots 
0:1,0:2 satisfying < oii < ^-J^ < 0:2 < 1, and there exists a unique uq G (0, ai) satisfying 

(2.6) hiuo) = hia2) 
or a unique ui G {02, 1) satisfying 

(2.7) h{ui) = h{ai). 
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Proof. Note that h{l) = 0, lim„_j.o+ ^(^) = +00, h'{u) > for n G (01,02) and h'{u) < 
for u G (0, ai) U (02, 1). Here oi < 0:2 are the roots for h'(u) = = 0. Then w = h{u) 
has a local minimum at ai and a local maximum at 02- Moreover, there exists a unique 
no G (0,ai) satisfying h{uo) = h{a2) or a unique ui G (02, 1) satisfying h{ui) = /i(qi) 
(see Fig. 2). □ 

Lemma 2.2. Under any one of the following cases: 

(i) B>1, 

(ii) < B <1, 27A> {1 - Bf, 

(iii) < i? < 1, 27A < (1 — B)^, u* < uq or u* > ui, where uq is given by (|2.6p . and 
ui by (I22D, 

we have {u — u*)[h{u) — h{u*)] < for < u < 1, where {u*,w*) = {u* ,h{u*)) is the 
positive equilibrium of (12. ip . 



By computing the eigenvalues at E'l, E and E* in sections 3, 4 and 5, we find that E, 
El, E* are all saddle points in the parameter ranges considered in this paper. The local 
unstable manifolds for E and Ei are two dimensional and the local stable manifold for E* 
is two dimensional. Generically, both type I and type II waves are transversal heteroclinic 
orbits. Therefore, we expect that both waves should exist for an open set of parameters, 
plus maybe some of its boundary points. 

We now state our main results. 



Theorem 2.1. Let /3 > A + B + 1 > 1, := (^^:^^:^ - ij and {u*,w*) be as in 

m- 

(1) IfO<c<c^:, then type II waves do exist while type I waves do not. 

(2) (i) If c > c*, A,B satisfy one of the three conditions in Lemma [27^ then type I 
waves do exit while type II waves do not. 

(ii) If c > c^:, < B < 1, 27 A < (1 — B)"^ and uq < u* < ui, then there is a 
traveling wave (j){s) with lim (/>(s) = (1,0) and (j){s) G ^(uw) — {iu,w)\0 < u < 
1,0 < w < /3(1 + c^/2)(l — u*),} for s > 0, while type II waves do not exist. 

(3) Let {u, w) be a type I or type II traveling wave. Then there exists a value 7* = 
^*{A,B,l3,c) such that if < ^ < 7*, {u,w) is non-oscillatory and approaches 
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(n*,w*) monotonically if s is sufficiently large, while if ^ > 7*, {u,w) have expo- 
nentially damped oscillations about {u*,w*) as s —>■ 00. 
Furthermore, we have 

l(02q\u } 

where Wc := ^ + c, 81 := -k{u*)q{u*), 82 := {2A + Bu*){l - u*) and 

Remark 2.1. T/ie results in \\.?>\ Theorem 2.2] is a special case in (2) of Theorem \2.1\ 
i.e. B = 0. Our contributions to this special case are as follows: (I) The existence of type 
I waves for cases c = c-t and c > c^:,u* = ui or u* < uq was not in that paper. (II) Type II 
waves and the bounded waves in our case B = 0, 27 A < (1 — B)^, uq < u* < ui were not 
discussed in [13\ . In fact, the latter cannot be obtained easily by the unbounded Wazewski 
set used in their work. (Ill) The non-monotone of traveling waves is not discussed in [13]. 

The proof of Theorem 12.11 will be given in the following three sections. In Section 3 and 
Section 4, we prove the assertion (1) and (2) respectively. The procedure of proofs will 
be divided into several steps with the main argument stated as several Lemmas for easy 
understanding. 

Consider the differential equation 

(2.8) y' = f(t,y), 

where f(t, y) is a continuous function defined on an open (t, y)-set Q. Let Qq be an open 
subset of 0, dQo be the boundary and CIq be the closure of Qq. 

Definition 2.1. [8, pp. 278] A point (to, 2/0) ^ ^ Ci dO,Q is called an egress point of ^Iq, 
with respect to (|2.8p . if for every solution y= y(t) of (j2.8p satisfying 

(2.9) y(to) = Vo, 

there is an e > such that (t, y{t)) E JIq for to — e < t < tQ. An egress point (to, Vo) ofVlQ 
is called a strict egress point of VLq if (t, y{t)) VLq for to < t < to + e with a small e > 0. 
The set of egress points of 0,q will be denoted by f^Q and the set of strict egress points by 
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r^Q"^. As a sufficient condition, the set of egress points I^q can be determined by verifying 
~^ ■ it > 0, where "7" is the vector field, and it is the outward normal vector of d^Q. 

If [/ is a topological space and V a subset of C/, a continuous mapping vr :[/—)• y 
defined on all of U is called a retraction of U onto V if the restriction 7r|y of ^ to F is 
the identity; i.e., 7r(n) G V for all u G [/ and ^[v) = v for all v £ V. When there exists a 
retraction of U onto V, V is called a retract of U. 

Lemma 2.3. [8^, pp. 279] Let f(t, y) be continuous on an open {t, y)-set Q with the property 
that an initial value determines a unique solution of (12. Sp . Let 0,q be an open subset of 
Q satisfying i7g = Qff^. Let S be a nonempty subset of 0,q U 0,q such that 5 H $7^ ™^ ^ 
retract of S but is a retract of I^q. Then there exists at least one point (to) 2/o) ^ S D^Iq 
such that the solution arc {t, y{t)) of (j2.8p - (12. 9p is contained in JIq on its right maximal 
interval of existence. 

3.. The existence of type I traveling waves 



At the equilibrium (1,0,0), ()2.3p becomes 
(3-1) "^(1,0,0) 



c c{A+B+l) ^ 



1 

lo 7(1-^) c] 
Then by (|3.ip . we see that the eigenvalues of (|2.2p at (1,0,0) are 



^ c- dc^ c+ -^l\A+B+i 

c 2 2 

If ^2 7^ Ms, then the eigenvectors corresponding to ^27^3 are 

1 , / -1 



= 1 + '=/^2, M2(l + C/i2)J , X3 = 1 + C/i3, ^3(1 + C^3 

Proof of the non-existence of the type I waves for < c < c*. . 

If < c < c*, then fi2 and ^3 are complex. We shall show that the heteroclinic orbit 
r of (12. 2p would oscillate around w = for large negative s. The heteroclinic solution F 
satisfies: 
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Let i := 7( ^_)_^_)_]^ — 1). Since /3 > A + S + 1, we know that £ > 0. In a small neighborhood 
of (1,0,0), using the polar coordinates w = psm{6), z = pcos{9), we obtain 

de ^ z^- cwz + 7wH a+b^u-,u^ - 1) 
ds p2 

Note c* = VaI. For any given c G (0, c*), let e be a positive number such that < e < 

2 

£ — ^. Since lim u{s) = 1, one can choose large Sc > such that 



£- e < 7(- „ , , „, , - 1) < ^ + e for s < -Sc 

^A + Bu(s) + u^(s) ' 



Therefore, for s < — Sc, we have 

— > cos2(0) - csin(e) cos{9) + {£ - e) sin2(e) 

(3.3) 

= [cos(0) - I sin(e)]2 + (£ - ^ - e) sin2(6'). 

The last expression is a periodic function of and is nonzero, which must be bounded 
below by a constant cq > 0. Therefore if s < —Sc, we have g| > cq > 0, which can leads 
to lim 6{s) = — oo. From w = ps'm(9), w{s) would be negative for some s < —Sc- This 

s— >— oo 

violates the requirement that the traveling wave solution considered should be nonnegative. 
The proof is complete. □ 

We now assume that c > c=k . Counting multiplicity, then there are three real eigenvalues 
satisfying /xi < < /U2 < If c > c*, then fj.2 < fJ-3, there are two eigenvectors (X2,X3) 
corresponding to (/i2,M3)- If c = c*, then fi2 = ps is a double eigenvalue. There are two 
generalized eigenvectors (X2,X3) corresponding to 1^2 = ps- By Theorems 6.1 and 6.2 in 
[SI pp. 242-244], there exists a two dimensional local unstable manifold W^^{Ei) tangent 
to the span of X2, X3. The points on Wl^^{Ei) can be represented by the local coordinates 
$2 : ^ K^, 

$2(m, n) = (1, 0, 0)'^ + m • X2 + n • X3 + o{\m\ + |n|). 

Consider a prism shaped solid W (see Fig. 3) in (u, w, z) space bounded by the following 
five surfaces: 

(1) The top surface := {{u,w^z) \ z = |t(;,0 < u < < w < Wm}, where 
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Namely, the quadrilateral ABCFG. 

(2) The bottom surface F;, := {(u, Wjz) \ z = —-^w, < u < 1,0 < w < Wm}- Namely, 
the quadrilateral ABCJH. 

(3) The front surface ¥f := {{u,w,z) \ w = K*{1 -u),u* <u < l,-^w < z < ^w}. 
Namely, the triangle CFDJ. 



(4) The right vertical surface := {{u,w,z) \ w = Wm,0 < u < u 



* 7 



Wm < Z < 



^Wm}- Namely, the quadrilateral GFDJHE, where one part GFDE is above the 
plane z = 0, and the other part EDJH is below the plane z = 0. 
(5) The back surface F^. := {(n, w,z) \ ti = 0, < w < Wm, —-^w < z < ^w}. Namely, 
the triangle AHG. 




u=l 



Figure 3. The graph of set 



Note that u* G (0, 1), thus we obtain 
K*(l - u*) 



^ {l + cd)[A + Bu* + {u*Y] l + cd , 

1 — n • -t; = w > w . 



(n*)2 



Next, we will prove the following lemma. 



Lemma 3.1. // an initial point Pq = iu{0),w{0), z{0)) is picking from the interior ofW, 
then the flow (j){s,Po){s > 0) of ([22]) can only exit W from Ft, ¥b, F^ n {z > 0} or the 
line segment GF. 



Proof. (1) The outward normal vector of F^ is nt = (0, — |, 1), and the vector field is 

uw \ ( _ fiv? 



1 



c \ A + Bu + u^ 



,Z,CZ + 'JW 1 



A + Bu + v? 
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Then it follows that 

7 = 



(3.4) 



> 



-w + 

-W + 'JW ( 1 



A+Bu+u^ J 
A+B+l 



W 



+ 7 1 



> 



by the assumption c > c*. Therefore, the top surface Ft belongs to the egress set. 
(2) The outward normal vector of is = (0, — ^, —1). Then it follows that 



-j:Z — cz — 'JW I 1 



A+Bu+u^ 



= i^)^ w + -fw^^§^ > 0, 
and this implies that the bottom surface F^ belongs to the egress set. 
(3) The outward normal vector of Fj is nj- = {K* , 1, 0). Then 



n/ 



■7 



< 



K* 
c 

K* 
c 

K* 



U 1 



U 1 



U 



c 

= K*{1 
< K*{1 
= K*(l 



A+Bu^ 




uw 




A+Bu^ 




uK*{l- 

A 1 D„, 1 


•> 



+ %w 



+ §K*(1 



K* 

c A+Bu+u'^ 
K* {u'f 



^ I c 

c A+Bu* + {u*y^ 2 



+ =i^*(l-n)(f-d) <0 
by the assumption d> ^. Therefore, the front surface Fj belongs to the ingress set. 

(4) The right vertical surface F^ is further divided into F^ = F^^ U F^j U "^rzi with 
F^i := F,. n {z > 0}, ■■= Fr n {z < 0}, F^g := F^ n {z = 0}. The outward normal vector 
of Ff, is = (0, 1,0). Then = z and this implies that F^^ belongs to the egress set 
while belongs to the ingress set. 

(5) It is easy to see that the flow of (12. 2p cannot enter or exit W through F^ since u = 
is invariant under the system (12. 2p . 

In order to check all the intersection part of two surfaces for the solid W, we make the 
following observation: 

// M and N are two surfaces that intersect along a line i, and if the vector field is 
transverse to these two surfaces, then the line £ is on the egress set if and only if the 
vector field points outward to both surfaces M and N. 
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From this observation, we see that the line segments GF is on the egress set. It is easy 
to check that F^g = n {z = 0} is not on the egress set. Furthermore, notice that AC is 
a part of u axis which is an invariant set of ()2.2p . which is not a part of the egress set. 

Thus, we have shown that F^, Fj,, F,,^ = F^ n {z > 0} and the hne segment GF belong 
to the egress set, and no more. The proof of the lemma is complete. □ 

Definition 3.1. Consider a two-dimensional surface S = span{\i\,\i2} in and let n 

he a normal vector to S. Any vector w ^ S is on the positive (or negative) side of S with 
respect to the normal n i/n • w > (or < 0). 

We say the two points wi and W2 are on the same side of S (or on the opposite side 
of S) if the product {n ■ wi)(n • \\r2) > (or <{)). 

In particular, if we choose n = ui x U2, then n • w = det(ui, U2, w). 



Lemma 3.2. There exists an arc C = C1C2 C W n Wl^^{Ei) of which one end point 
Ci G Ff, and the other end point C2 G F;,. 



Proof. First, assume that c > c*. Then Ei has two distinct positive eigenvalues. The 
two dimensional unstable manifold is tangent to S" = span(X2,X3) = span(X2, — X3). 
Recall that Ci^ = (u* — l,tfmj f'U^m)^ and (jJ = {u* — l,Wm,—-l:Wm)'^ are the vectors 
corresponding to the rays GF and CJ, and u = (1, 0, 0)-^ be the unit vector on the u— axis 
with a start point C = (1,0,0). 

Note that I3>A + B + I,c>c^: and d > |, we obtain 

det(X2,X3, Cp) = ^fff:^ (1 + f ) (m3 - ^2) - (1 - n*)(l + c/U2)(l + c^3)(/X3 - M2) 



= (1 - «*)(/^3 - /i2) 
= (1 - ^^*)(M3 - ^2) 



K* 
A+B+1 

{l+cd)l3 
A+B+1 

(1+C- 



2 7 V ^ + 2 



(1 + C/i2)(l + C/U3) 



l-c2-c2.% 



det(X2,X3,'u) = (^3 - /i2)(l + c/i2)(l + C/U3) > 0, 



det(X2,-X3,C:}) = -{l-U*){fl3-fl2) 



K* 



- (1 + 7 + C^) - (1 + C//2)(l + C//3) 



<0, 



A + B + 

det(X2, -X3, u) = - det(X2, X3, -u) < 0. 

This implies that u and Cp are on the same side of the tangent plane S. Also, u and 
CT} are on the same side of S. 
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Next assume that c = c*. Then Ei has a double real eigenvalue iJ,2 = H3 



f. The 



two dimensional unstable manifold is tangent to = span(X2, X3), where X2 and X3 are 
generalized eigenvectors corresponding to //2- Denote the first row of the matrix 






{7^1 - J{i,o,o))^ 



c{A+B+l) ^ 2{A+B+1) c{A+B+l) 





V 











by ri. Since any generalized eigenvector X satisfies {p2l ~ J(i,o,o))'^^ = 0; '^^ see that 
ri is a normal vector to S. Then for any G M^, whether v is on which side of S can 
be determined by the sign of the inner product ri • v. Recall that u = (1, 0, 0)-^ and 
^ = (— 1) i)'^) it is easily checked 



ri ■ u 



ri • 



2 ^ c 



+ 



1 



K* 



2{A + B + 1) 



c{A + B + 1) 2{A + B + 1) 

,c 1 K*-(A + D + 1)(1 + ^) 
4 c^' c{A + B + l)K* 

Since K* = {1 + cd){-^ + f + 1) > (1 + cd){A + S + 1) > (1 + ^){A + B + 1), we have 
ri • > 0. 



Next recall that 



Wm 



( L 1 



7\T 



. It is easily verified 



ri > ri > 0. 

In conclusion, u, are on the same side of S just like the case c > c*. Denote 

this side as the "positive-u side of S"' . 

In geometry, same notations are often used for vectors and rays. In the rest of the 
section, we assume {CTP, c}, u) arc three rays with vertex C . Let co{CJp, cJ, u) be the 
convex hull generated by the rays C^, CjJ and the ray pointing to the positive «— axis 
with the starting point C. This is an infinite cone of triangular cross section and with the 
vertex C. Then except for the point C, co(C^, C^, u) is on the "positive-n side" of the 
tangent plane S. 
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Define 

Si := {(u,w,z) I z = -w}, S2 ■■= {(u,w,z) I z = w}. 

2 c 

Note that ¥t (Fb) is a part of Si (52). 

Note tlie plane S intersects with Si transversely and S is the tangent plane of Wl^^{Ei) 
at El, the unstable manifold W^^^^Ei) intersects with Si on a smooth line segment ii. 
If we make ii sufficiently short, then £1 does not enter co{CF,CJ,u). Denote by £t the 
part of ii where z > and which is on F^. We now select Ci on if It is clear that 
Ci e Wf'^^iEi) n ¥t but not in co{CF, cJ, u). 

Similarly, by using the property of 6*2, we can construct a short line segment which 
is on W^^JyEi) n Ffc but not in co{Cp,C~},u). Select a point C2 G ib- Using the local 



coordinates u = u{w,z) for Wl^^{Ei), we can construct a curve segment C = C1C2 on 
Wl^^{Ei), connecting Ci and C2 and is between the two surfaces Ft and Ff,. 
Let d{Pi, P2) be the distance function in M^. Define the minimum distance 

rj := mm{d{Pi,P2) \ CPi G co{CF ,0^ ,u),\\CPi\\ = 1,P2 G S]. 

Then < r/ < 1. If C(^ is a nonzero vector in co(Cp, cJ, u), then the distance d{Q, S) > 
rjp, where p = ||CQ||. On the other hand, since is tangent to S, the distance of 



any point Q G C1C2 to is 0{p ). If p is sufficiently small, 0{p ) < rjp. Based on this, 



we find that the entire curve C1C2 is inside W if its distance to the vertex C is sufficiently 
small. □ 

Lemma 3.3. There is a point Pq = {uo,wo,zo) G C fl W such that the flow (j){s,Po) will 
remain in W for all s > 0. 



Proof. Note that A + Bu + u^ = has roots "^^^.f" Set 



u 



-00, -4A< 0. 

Define P = {u,oo), then u',z' has no sing ularity on P. Let n = V x and = W. 
Then = nff' = Ft U (F^ D {z > 0}) UGFU F^. Also, Ci U C2 is not a retract of C but a 
retract of fi^- Lemma 12.31 there is a point Pq G C n W such that the flow 0(s, Pq) will 
remain in W for all s > 0. □ 
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Lemma 3.4. Let Pq be defined as in Lemma [STSi Then (j){s,Po) — >■ {u*,w*,0)'^ as s 



+00. 



Proof. Let u be the negative root of the equation 1 — = 0, then the two roots 

of 1 - ^-^V^ = are u* and u . Define a Liapunov function 



V(u,w,z)=c'y / at + lew — w — z\ + w clog — 

Ju* IW W* 

It is easy to check that V{u, w, z) is continuously differentiable and bounded below on the 
compact set W. Moreover, 

_ c"i{u-u*){u-u-) u ( _ „, _ uw \ j_ („ _ Wz _ aw^\ 
' V w 



•y{u—u*)(u—u ) I 



2 



A+Bu+u'^ 



where /i(n) is defined in Section 2 and t/;* = h{u*). By Lemma 12.21 it immediately 
follows that (n — u*)[h{u) — w*] < for < u < 1. Furthermore, ^ = if and only if 
{u = u*,0 < w < Wm,z = 0}. The largest invariant subset of this line segment in W is 
the positive equilibrium {u*,w*,0). By the LaSalle's invariance principle [12], it follows 

that (j){s, Pq) ^ {u*,w*, 0)"^ as s ^ +00. □ 

Proof of existence of traveling waves for c> c^,. Assume that one of the conditions 
in Lemma 12.21 holds. Choose a point Pq in C n W as defined by Lemma 13.31 By Lemmas 
I3.3fl3.41 we see that (f){s, Pq) will remain in W and further approach the positive equilibrium 
{u*,w*,0)'^. Also, (/>(s,Po) (1,0,0)'^ as s ^ -00 since Pq G Wl'^^iEi). Thus, a type I 
traveling wave solution has been constructed for c> c^,. 

If < P < 1, 27A < (1 - Bf, uo <u* < ui, then by LemmaESl there is a traveling 
wave </>(s,Po) with lim (/)(s,Po) = (1,0,0) and 0(s,Po) € W for s > 0. Note that the 

s— 00 

projection of W on (n, w) plane is in the rectangle W(„ „,) := {{u,w)\0 < u < 1,0 < w < 
K*{1 — u*),}, where K* = /3(1 + cd), d is any constant larger than c/2. The proof has 
been completed. □ 
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4.. The existence of type II traveling waves 



At the equilibrium E = (0, 0, 0), (l23|) 



becomes 



/i 







c 



(4.1) 




1 



\0 7 c) 



Then the eigenvalues of (j2.2p at are 



_ c - v^c^ + 47 



< 0, = - > 0, 



C+ a/c2 + 47 



> 0. 



c 



2 



Thus there is a two dimensional local unstable manifold W1^^^{E) based at E. The eigen- 
vectors corresponding to 2^2, 1^3 are respectively Y2 = (1,0,0)-^,Y3 = (0,1, z^s)-^. Let 
L = (0,1,0)"^ be a vector that is complementary to the plane S = span{Y2,Y3}. In a 
small neighborhood of E^ the points on Wf^^^E) can be expressed as: 



where £*(m, n) = 0{m? + n^) is a smooth function of (m, n). Since the u-axis is invariant 
under the flow of ([2^2]) . if m is small, then mY2 G W,"^(i?), that is, r(m, 0) = 0. 

On the other hand, equation ()2.2I) is linear if u = 0. For any n G M, the line nYs 
is invariant under the flow and nY3 G Wl^^{E). This shows i*{0,n) = 0. Based on 
i*{m,0) = i*{0,n) = 0, we have a better estimate i*{m,n) = 0{\mn\). And any point on 
^loci^) expressed as 

(4.2) Q(m,n) = {u,w, z f = (0, 0, 0)^ + m • Y2 + n • Y3 + (0(|mn|)) • L. 

Proof of the non-existence of the type II waves for c > c=k. We present an indirect 
proof. If there is a heteroclinic solution {y(s) = {u{s),w{s),z{s))'^ : s G M} connecting E 
to E* , then for sufficiently large negative si, we have y(si) G Wl^^{E). Let (m{s),n{s)) 
be the parameter representation for y(s) as in (14. 2p . Then we claim that n{si) > 0. For 
from (j4.2p . if n(si) < 0, then ii)(si) < 0, which is meaningless in the biological context. If 
n(si) = 0, then w{si) = z{si) = 0, thus y(si) is on the u-axis. However, the u-axis is an 
invariant manifold connecting E to Ei, this is a contradiction. 

Using n(si) > 0, we have z{si) = U3w{si) + o{\n\) > |iz;(si). We can show that 
z{s) > §w{s) and wi^s) > for all s > si. Here is the proof. 



(u, w, zf = (0, 0, 0)^ + m • Y2 + n • Y3 + r(m, n)L, 
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Consider the open set A := {0 < u < l,w > 0, z > §w}. Let = z{s) — ^w{s). 
From (j2.2p . we calculate x'i^) in A: 



x'{s) = cz{s) + 'yw{s) 1 



/3n2(s) 1 c 





A + Bu{s) + u'^{s)\ 2 



= 0. 



Now the heteroclinic solution y(s) satisfies y(si) G A and < u{s) < 1 for all s. We 
can prove y(s) G A for all s > si by contradiction. Assume that S2 > si is the first time 
that y(s) hits the boundary of A. Then either (1) 10(32) = or (2) x{s2) = 0. Case (1) is 
impossible since for si < s < S2, w'{s) = z{s) > in A. Case (2) is also impossible since 
for si < s < x'is) > which leads to xis2) > xi^i) > 0- 

However, if y(s) G A for s > si, then lims^oo y('5) 7^ E* . The proof has been completed. 



The existence of the type II waves for < c < c*. Define a solid Wq which 
is a modification of W as in §3, Fig 3. Compared to W, the top surface is replaced by 
¥t := {{u, w,z) I < u < 1, < li; < Wm, z = dw} where d > 1/3; the right vertical surface 
is replaced by F,. := {(u,w,z) | < n < u*,w = Wm, —-^Wm < z < dwm}', and the front 
surface is replaced by Fj := {{u,w,z) \ u* < u < l,w = K*{1 — u),—^w < z < dw}. The 
back surface is F^ := {(n, Wjz) \ u = 0,0 < w < Wm, —"^w < z < dw}. 

The bottom surface is unchanged. 

Lemma 4.1. The egress sets for Wq are the bottom surface Fj, and part of the right vertical 
surface F^ n {z > 0}. 

Proof Notice that {u = 0} is an invariant set thus the back surface F^ is not an egress 
set. 

The set Fr n {z > 0} is obviously an egress set due to w' = z > there. 
The bottom surface F^ remains the same and hence is an egress set. 



□ 
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The front surface is larger than that of W. However its normal vector remains the 
same. Modifying the estimates in the proof of Lemma 3.1, we have 



^ ^ A+Bu+u'^ 



+ dK*{l-u) 
< K*{l-u) [l-l±^ + d]=0. 
Thus, is still an ingress set. 

Since < d, similar to the proof of Lemma 3.1, we have 



. / =-dz + CZ + -iW[l- ■ 2 

\ A + Bu + w^J 
( 

={-d + c)dw + ^w 1^1- ^^^^^^, 



-d^ + cd + 7 ( 1 



A + Bu + v? 



< — w{d^ — cd — 

< — w{i>^ — cus — 7) = 0. 

Hence, Fj belongs to the ingress set. 

Whether the edges of Wq is an egress set can be checked easily. In particular, the line 
GF as in Fig. 3 is not an egress set. 

n 

In order to use Lemmma l2.3l we shall construct a curve £ C W^{E) n Wq, of which the 
two end points belong to the two disjoint egress sets of Wq. 

Lemma 4.2. Assume that < c < c*, then there exists an Pi £ Wl^^{E) n such that 
the flow (j){s, Pi) enters Q = {{u, w , z)\u > u* , w < w* , z < 0} for some finite s = s, where 
Q = {{u,w,z)\u >0,w>0,z> 0}. 

Proof. The proof is similar to that Dunbar in Lemma 10 [2], and shall be skipped. □ 

Choose a constant w such that w{s) < w < w* . Define a solid as W2 := co{B, C, D, J)n 
{w < w}. 

Lemma 4.3. The only egress set for W2 is the egress set Fj for Wq. 
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Proof. The surface w = w, z < 0, where w' < 0, cannot be an egress set for W2. 

The surface CD J is part of for Wq thus it cannot be an egress for W2. 

The surface BDJ n {w < w} cannot be an egress set since u' = at (u = u* ,w = w*) 
from the first equation of (|2.2p . We now have u = u* but w < w* , therefore u' > there. 

On the surface BCD n{w <w}, z = 0. From ([22]), z'{s) = (l - xfr^p^) < 0' 
since it would be if n = u*. But now we have u > u* . 

□ 

Proof of the existence of the type II waves for < c < c* . Let Pi be the point as 
in Lemma |4.2[ The orbit </>(s,Pi) enters Wq at some s = S3 > s. Let P3 = (j){s3,Pi). The 
flow P3) cannot stay in W2 for all s > S3 because w'{s) < 0, z'{s) < there. Therefore 
it must exit through its egress set F;, as from lemma HTSl 

On the other hand, from (j4.2p . for a small real n > 0, (5(0, n) := nYs is on Wj^^{E). 
Moreover, from ()2.2p . the solution (l){s,Q{0,n)) hits the surface w = Wm transversely in 
finite time. From the expression (j4.2p . there is a small m > such that the corresponding 
Q{m,n) is near Q{0,n) and its u-coordinate is small and positive. Thus, Q{m,n) G 
Wl^^{E) n Wo and the flow (j){s,Q{m,n)) hits the surface w = Wm in finite time and 
its n-coordinate is small and positive. Let P2 = Q{m,n). Then (p{s,P2) exits Wq at 

F,. n{z > 0}. 

With the given end points Pi and P2 and using the local coordinates for we 
can construct a small curve P1P2 C Wl^^{E) n Wq. Let S4 be the first time that 4'{s,Pi) 
hits Ffe and let S5 be the first time that (p{s, P2) hits F^ H {z > 0}. Then the curve £ can 
be obtained as the union of three curves 



£ := {<i){s, Pi) : < s < 54} U P1P2 U {cp{s, P2) : < s < S5}. 



From Lemma [2.3^ there exits a point Pq G P1P2 such that the flow (/)(s,Po) remains 
in Wo for s > 0. By using the same Liapunov function as in Lemma 13.41 we see that 
(/>(s,Po) E* as s ^ 00. The proof has been completed. 

□ 



(5.1 



traveling wave solutions for a predator-prey system 
5.. Oscillation of the traveling wave solutions 

Evaluating the Jacobian matrix of (|2.2p at E* = {u*,w*,0), we have 

/ ^k{u*)q{u*) -l{u*yq{u*) 0^ 

1 

\-P-f{2A + Bu*){l-u*)qiu*) c/ 
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J, 



where k{u) = —2v? + (1 — B)u^ — A (see Lemma [2?T]) and q{u) = xkB^ITu^' Observe that 



f3{u*) q{u*) = 1. Using 7 as a parameter, the characteristic polynomial of J(„ 



{u* ,w* ,0) 



IS 



p(A,7) = A 



A 



k{u*)q{u* 



(c - A) + ^{2A + Bu*){l - u*)q{u*) 
c 



Note that if B > 1, then k{u*) < 0. If < B < 1, A > 0, then k{u*) < also holds. If 
< i? < 1, A < 0, < ti* < no < ai and a2 < ui < u* < 1, we have k{u*) < (see page 
6 for details) and thus -k{u*) > 0. Let 5i := -k{u*)q{u*) and 82 := {2A + Bu*){l - u*). 
Then (5i > 0, (52 > and 



+ c A^ + JiA + ^52q{u* 



-Si 



+ c] X + 61, 



Proof of (3) in Theorem \2.1[ Based on 

(5.2) p'(A,7) = -3A2 + 2 

we find that p'{0,j) > and p'{X,^) has two critical points A_ < < A+. Together with 
the fact that on the real line, p(A,7) — )■ ^00 if A — )■ ±00, we find that when 7 = 0, the 
graph of p(A, 0) is an "S" shaped function passing through the origin. When 7 > 0, the 
graph is a shift up by ^62q{u*) to that of p(A,0) (See Fig. 4). 

P(A,7) 




Figure 4. The graph of p(A,7) for 7 > 
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Therefore, if < 7, the equihbrium point E* is hyperbohc, with a two dimensional 
Wf^^{E*) and a one dimensional W;^^(i?*). 

From Fig. 4, it is easily seen that there is a threshold value 7* = {A, B, (3,c). If 
< 7 < 7*, there are two distinct negative real eigenvalues for E* . If 7 = 7*, there 
is a repeated negative real eigenvalue. If 7 > 7*, there is a complex conjugate pair of 
eigenvalues with negative real part. 

Hence, if < 7 < 7*, -E* has the two real negative eigenvalues. Let y{s) be a solution 
to (j2.2p that is on Wf^^{E*). Then y(s) approaches E* monotonically if s is sufficiently 
large. If 7 > 7*, y(s) approaches E* with damped oscillations as s — >■ 00. 

The negative root A_ of (|5.2|) is 

A- = ^ (uic - V^c+ ^^1) 
with Uc = + c. There exists a unique 7* such that 

p(A_,7*) = -(A_)3 + L^,(A_)2 + <5iA_ + ^62q{u*) = 0. 

c 

Using p'(A-,7*) = to simplify p(A_,7*) (long division), the remainder is a first order 
polynomial in A_. Some calculations show that 

This completes the proof of (3) in Theorem 12.11 

□ 
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